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PT5 Background 


Motivation 

— Build upon previous human-in-the-loop simulations results and lessons learned to 
identify minimum DAA display and guidance requirements for draft SC228 MOPS 

• PT4 

- A suite of integrated guidance tools led to faster pilot responses, fewer losses of 
well clear and less severe losses of well clear when they did occur 

• iHITL 

- Integrated guidance tools led to less severe losses of well clear and faster pilot 
responses than seen in PT4 

• AFRL Maneuver Study 

- Guidance (in the form of 'banding') led to faster pilots responses and fewer 
collision avoidance alerts 


PT5 Background 


• Modifications from previous sims: 

- Guidance tools were no longer tightly coupled to the ground control station's auto 
pilot interface 

- Removed advanced features present in iHITL (e.g., well clear ring & dead reckoning 
lines) 

- Modeled sensor uncertainty for the first time 

• Critical to test displays and algorithms with 'imperfect' data prior to flight tests 

- Implemented alerting structure as part of the draft MOPS 

- Increased workload on the pilot 

• More secondary tasks and interaction with their route 
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Experimental Design 


Mixed Factorial Design 

- Display Configuration (Within-Subjects Independent Variable): 

• Configuration 1: Minimum Information Set (No Guidance) 

• Configuration 2: Stratway+ No Fly Bands 

• Configuration 3: JADEM Omni Bands 

• Configuration 4: JADEM Vector Planning Tools 

- Sensor Performance (Between-Subjects Independent Variable) 

• Level 1: Perfect Surveillance Data 

• Level 2: Imperfect Surveillance Data 

Embedded Variable 

- Intruder Equipage (manipulated within each scenario) 

• Transponder-equipped (detected via UAS's ADS-B) 

• No Transponder (detected via UAS's on-board RADAR) 


Experimental Design 


Participants 

- 16 active UAS pilots 

• Avg. 37 years old (all male) 

• Manned Flying Experience 

- Civil airspace: 575 avg. hrs 

- Military: 1760 avg. hrs 

• Unmanned Flying Experience 

- Civil airspace: 30 avg. hrs 

- Military: 1100 avg. hrs 

Task 

- Fly simulated MQ-9 through Class E airspace (Oakland Center - ZOA 40/41) 

• Navigate along pre-filed routes (used AFRL's Vigilant Spirit Control Station) 

• Maintain well clear from pre-scripted conflicts 

• Coordinate with ATC 

• Attend to secondary tasks (e.g v chat messages, system alerts) 



Experimental Design 


Pre-planned conflicts with ownship 

- 6 scripted encounters predicted to lose well clear 

• 3 with cooperative traffic (detected at max range of 15nm) 

• 3 with non-cooperative traffic (detected at max range of 8nm, with limited FoR) 

- 3 scripted encounters predicted to become preventive self separation alerts 

Dependent measures 

- Losses of well clear 

• Rates, severity and type 

- Pilot measured response 

• Initial & total response times 

• Communication and edit times 

- Subjective Feedback 

• Post-Trial/Simulation questionnaires, debrief 


Experimental Design 


• Simulation confederates 

— NATCA controller managed UAS and manned traffic within ZOA 40/41 
• Simulated manned traffic based on actual sector activity 

- Pseudo-pilots managed all manned traffic to provide dynamic sector activity 

- ATC SME operated as 'ghost' controller to ensure conflicts were generated 



r 

Air Traffic Control Station (MACS) 
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Draft MOPS Alerting Structure 
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Name 

Pilot Action 

Buffered Well 
Clear Criteria 

Time to Loss of Well 
Clear 

Aural Alert 
Verbiage 

Self Separation 
Warning Alert 

• Immediate action required 

• Notify ATC as soon as practicable after 
taking action 

DMOD = 0.75 nmi 
HMD = 0.75 nmi 
ZTHR = 450 ft 
modTau = 35 sec 

25 sec 

(TCPA approximate: 
60 sec) 

"Traffic, 

Maneuver 

Now" 

Corrective Self 
Separation Alert 

• On current course, corrective action 
required 

• Coordinate with ATC to determine an 
appropriate maneuver 

DMOD = 0.75 nmi 
HMD =0.75 nmi 
ZTHR = 450 ft 
modTau = 35 sec 

75 sec 

(TCPA approximate: 
110 sec) 

"Traffic, 

Separate" 

Preventive Self 
Separation Alert 

• On current course, corrective action 
should not be required 

• Monitor for intruder course changes 

• Talk with ATC if desired 

DMOD = 0.75 nmi 
HMD = 1.0 nmi 
ZTHR = 700 ft 
modTau = 35 sec 

75 sec 

(TCPA approximate: 
110 sec) 

"Traffic, 

Monitor" 

Self Separation 
Proximate Alert 

• Monitor target for potential increase in 
threat level 

DMOD = 0.75 nmi 
HMD = 1.5 nmi 
ZTHR = 1200 ft 
modTau = 35s 

85 sec 

(TCPA approximate: 
120 sec) 

N/A 

None (Target) 

• No action expected 

Within surveillance 
field of regard 

X 

N/A 










Display Conditions 
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Display Conditions 
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LOSS OF WELL CLEAR DATA 

• LoWC Rate (by Display) 

• LoWC Categories 

• LoWC Severity (by Display) 


Actual vs. Predicted Losses of Well Clear 
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Info Stratway+ Omni Bands Vector Planner 


Configuration 


Did the pilot's ability to prevent losses of well clear differ by display? 

- Info Only (19.8%) was roughly four times as likely as Stratway+ (6.5%) and Omni 
Bands (4.2%) to result in LoWC, a significant difference (p<. 05) 

• Info Only was roughly two times as likely as Vector Planner (10.3%) to lead to LoWC, 
which approached significance (p=. 086) 

- No significant differences seen between the three guidance displays in terms of 
LoWC 





LoWC by Pilot 


Statistical 

Significance 

Info Only 

Stratway+ 

OmniBands 

Vector Plan. 

Info Only 

1 

0.016 

0.002 

0.087 

Stratway+ 

0.016 

1 

0.256 

0.560 

OmniBands 

0.003 

0.260 

1 

0.123 

Vector Plan. 

0.087 

0.557 

0.123 

1 



Number of LoWCs by Pilot Number 
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0 
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Pilots 2, 6 & 13 had larger number of LoWC 
Pilots 7, 11, 16 had zero LoWC 








Unintentional I Pilot Responsibility 


Categories of LoWC 


• Assigned each LoWC one of the following categories: 


Code 

Type 

Description 

l 

Ineffective 

Maneuver 

Initial maneuver from traffic was not sufficient to prevent LoWC, even though 
algorithm was providing accurate information 

2 

Too Slow 

Pilot did not initiate the maneuver quickly enough (i.e., upload was not sent prior 
to LoWC or sent very late), even though algorithm was providing accurate 
information 

3 

Too Early 
Return 

Pilot returned to course too early, causing a LoWC with an aircraft they had 
already successfully avoided 

7 

Diverging 

LoWC 

LoWC occurred when ownship and threat were diverging, no visual or aural alert 
was issued to pilot 

Code 

Type 

Description 

4 

Late Intruder 
Acceleration 

State change by the intruder, made with less than 15sec to LoWC, escalated the 
severity of the encounter 

5 

Elevation 

Error 

Intruder (in truth) was level 500ft above or below ownship; altitude error caused 
LoWC 

8 

Vigilant 

Spirit 

Malfunction 

Bug when Vigilant Spirit was in altitude override mode resulted in the visual/aural 
alerting failing to issue or issue late 










Number of Encounters 


LoWC by Category and Time to LoWC at First Alert 
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LoWC Severity 


• All Displays: 

- Actual Separation / Separation Threshold 

- Less than 1 = spatial separation was NOT maintained 
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MEASURED RESPONSE DATA 

* Total Response Times (by Display) 

* Comparison to Previous Sims 

* Total Response Times (by Additional Variables) 


Encounter Statistics 


• Total Number of Encounters: 

- 282 encounters analyzed in following measured response data 

• 37 (13%) encounters were excluded due to pilot maneuvering prior to receiving an alert 

• By Threat Level 

- 244 Corrective SS Alerts issued 

- Ill SS Warning Alerts issued 

• By Intruder Equipage 

- 138 encounters with Cooperative Traffic 

- 144 encounters with Non-Cooperative Traffic 
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Measured Response Timeline & Associated Metrics 
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Measured Response Timeline & Associated Metrics 
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Info Only 


Stratway+ Omni Bands 
Configuration 


Vector 

Planner 


• Configuration had a significant effect on Total RTs: 

- Total RTs for Stratway+ were 8.2s shorter, on average, than those in Info Only, a 
significant difference (p= 0.02) 

- No other configurations differed significantly 

• On average, pilots took 22.8s to upload a final maneuver following the onset 
of a Corrective SS or SS Warning alert 

- 50% within 19s, 90% within 42s 


24 






Total Response Times Across Simulations 
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PT4- Total Response Times 




• 0 

Aircraft well Clear 

Maneuver Time Threshold 

0 


Basic Integrated (45s) 30 sec 

^ 0 

Basic Standalone (39s) 36 sec 

0 — 
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Advanced Integrated (32s) 43 sec 


iHITL — Total Response Times 
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PT5 - Total Response Times 
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Info Only Stratway+ Omni Bands Vector Planner 
Configuration 


• Pilots responded, on average, 10s faster to SS Warning Alerts than they did to 
Corrective SS Alerts 


- Pilots exhibited less variability between displays when responding to SS Warning 
Alerts than to Corrective SS Alerts 

• Range for SS Warning Alerts: 11s - 15s 

• Range for Corrective SS Alerts: 19s - 30s 
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Perfect Noise 


• Pilots responded, on average, 4.5s faster to non-cooperative traffic than they 
did to cooperative traffic, which was a significant difference (p=. 008) 

- There was also less variability in pilots' responses to Non-Cooperative encounters 

• Sensor model was not found to have any effect on pilot's Total RTs 
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Question 4: 

DID PILOTS DIFFER IN THEIR INITIAL RESPONSE TO 
ENCOUNTERS BASED ON THREAT LEVEL? 


Measured Response Timeline & Associated Metrics 
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Notify vs Upload Time : 
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Stratway+ Omni Bands Vector Planner 
Configuration 


• Pilots, on average, initiated their edit 8.6s faster in response to SS Warning 
Alerts than they did to Corrective SS Alerts 

- There was also less variability in general and between displays when pilots were 
responding to SS Warning Alerts 

- Pilots responding to SS Warning Alerts within 5s in all four displays 
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SUBJECTIVE DATA 

• Workload Ratings (by Display) 

• Post-Trial Response (by Display) 

• Post-Sim Response (by Additional Variables) 


Workload 


Omni Bands (D3) resulted in significantly lower workload ratings than the 
Vector Planner (D4) for 5 of the 6 scales: 

- Mental, Temporal, Effort, Frustration, & Performance Degradation 

• Only Physical Demands failed to result in a significant difference 

— Stratway+ and Omni Bands were never rated as significantly different 
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Post-Trial 


Did the display provide the necessary information to perform a maneuver 
for separation? 

— Stratway+ and Omni Bands were rated as significantly better at providing 
sufficient information 
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Info Only Stratway+ Omni Bands Vector Planner 


Post-Sim & Debrief 


All 3 alert levels were rated favorably by pilots 

- Overwhelmingly agreed visual display of alerts were easy to understand and 
distinct 

- Agreed that auditory alerts were easy to distinguish and useful 

- Reported that the timing of the alerts were appropriate 

Information level 

- Pilots generally agreed that they had all the information needed to perform the 
task 

- Several commented that a dead-reckoning line or 'history trail' would be useful 


Summary 


• Suggestive Guidance: 

- Suggestive guidance in the form of banding resulted in safer and more timely 
maneuvers away from conflicts 

• Fewer overall number of LoWC for both banding displays 

- Pilots responsible for significantly fewer (75% fewer) LoWC 

• Less severe LoWC for both banding displays 

- No instance of intruder penetrating more than 40% of the well clear threshold for 
Stratway or Omni Bands 

• Shorter Total RTs for both banding displays 

- Stratway+ was significantly shorter than Info Only but both banding displays 
showed quick responses 

• Pilots self-report as preferring the banding displays 

- Pilots reported that Omni Bands significantly reduced pilot workload relative to 
Vector Planner, but both banding displays showed lower workload ratings in 
general that Info Only and Vector Planner 

- Both banding displays were judged to provide valuable information not present in 
the other conditions 
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Summary 


Alerting Logic: 

- Subjective feedback was overwhelmingly positive 

• Visual and auditory features were rated highly 

• Pilots understood the alerting structure with no suggested changes 

- SS Warning alert resulted in pilots more quickly uploading a maneuver to the 
aircraft with less variation between displays (by 10s) 

Categorization of LoWC 

- The categories of LoWC developed here revealed several sources of LoWC not 
previously considered: 

• Numerous LoWC occurred when ownship and threat were diverging 

• Numerous LoWC also occurred due to the pilot returning to route too soon 
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Going Forward 


Flight Test 3 (presented by Debra yesterday) 

— Use flight test environment to verify the results we saw in PT5 

• Flying surrogate aircraft 

- Brining 2 displays from PT5 

• Information Only 

• Omni Bands 


BACK UP SLIDES 


Alerting Logic 


> Sensor Ranges 

• Simulated cooperative sensor: ADS-R/TCAS-like ranges 

• Lateral Range: 15 nm 

• Vertical Range: +/- 5000 ft 

• Simulated non-cooperative sensor: based on state-of-the-art 
airborne RADAR 

• Lateral Range: 8 nm 

• Azimuth: +/- 110 degrees Surveillance Range 

• Ele 
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Parameters for Noisy Cooperative Sensor 


Noisy Cooperative Sensor ("Transponder") 


Parameter 

Value 

Unit 

Field Of Regard 

Range 

15 

nmi 

Azimuth 

360 

deg 

Elevation 

+/-90 

deg 

Accuracy 

Range Error Mean 

0 

nmi 

Range Error Std. Dev 

0 

nmi 

Range Moving Avg. Window Size 

1 

measures 

Azimuth Error Mean 

0 

deg 

Azimuth Error Std. Dev 

2 

deg 

Azimuth Moving Avg. Window Size 

3 

measures 

Altitude Quantization 

100 

feet 

Altitude Moving Avg. Window 

6 

measure 


Yellow denotes the noise model variables that will used for PT5. 
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Parameters for Noisy Non-Cooperative Sensor 


Noisy Non-Cooperative Sensor ("Airborne Radar") 



Parameter 

Value 

Unit 

Field Of Regard 

Range 

6 

nmi 

Azimuth 

+/-110 

deg 

Elevation 

+/-20 

deg 

Accuracy 

Range Error Mean 

0.008 

nmi 

Range Error Std. Dev. 

0.001 

nmi 

Range Moving Avg. Window Size 

1 

measures 

Azimuth Error Mean 

0 

deg 

Azimuth Error Std. Dev. 

2 

deg 

Azimuth Moving Avg. Window Size 

3 

measures 

Elevation Error Mean 

1 

deg 

Elevation Error Std. Dev. 

1 

deg 

Elevation Moving Avg. Window Size 

6 

measure 


Yellow denotes the noise model variables that will used for PT5. 


Parameters for "Perfect" Cooperative Sensor 


"Perfect" Cooperative Sensor ("ADS-B") 


Parameter 

Value 

Unit 

Field Of Regard 

Range 

15 

nmi 

Azimuth 

360 

deg 

Elevation 

+/-90 

deg 

Accuracy 

Latitude Error 

0 

deg 

Latitude Error Std. Dev. 

0 

deg 

Latitude Moving Avg. Window Size 

1 

measures 

Longitude Error 

0 

deg 

Longitude Error Std. Dev. 

0 

deg 

Longitude Moving Avg. Window Size 

1 

measures 

Altitude Error 

0 

deg 

Altitude Error Std. Dev. 

0 

deg 

Altitude Moving Avg. Window Size 

1 

measure 
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Parameters for "Perfect" Non-Cooperative Sensor 


"Perfect" Non-Cooperative Sensor ("Perfect Airborne Radar") 



Parameter 

Value 

Unit 

Field Of Regard 

Range 

6 

nmi 

Azimuth 

+/-110 

deg 

Elevation 

+/-20 

deg 

Accuracy 

Range Error Mean 

0 

nmi 

Range Error Std. Dev. 

0 

nmi 

Range Moving Avg. Window Size 

1 

measures 

Azimuth Error Mean 

0 

deg 

Azimuth Error Std. Dev. 

0 

deg 

Azimuth Moving Avg. Window Size 

1 

measures 

Elevation Error Mean 

0 

deg 

Elevation Error Std. Dev. 

0 

deg 

Elevation Moving Avg. Window Size 

1 

measure 
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> 50% of pilots notified ATC within 12 seconds 
of the onset of the alert 

> 90% of pilots notified ATC within 33 seconds 
of the onset of the alert 
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Info Stratway+ Omni Bands Vector Planner 

Configuration 


> Pilots on average notified ATC 7.39 seconds 
faster following an SS Warning alert than a 
Corrective SS alert 

■ Median Difference = 1 second faster in 
response to Corrective SS alert 


> Unable to test for significant interaction 

■ Much larger differences between Corrective 
Notify times and Warning Notify times for D1 
and D4 
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Perfect Noise 


> Th e r e was no significant e ff e ct of Disp l ay on 
I n i tia l R e spons e T i m e for a ll SS a le rts, p > .05 

> Pilots on average notified ATC 1.16 seconds 
faster in response to alerts to cooperative 
aircraft 

■ Median Difference = 4 seconds faster in response 
to cooperative traffic 


> Th e r e was no significant e ff e ct of Disp l ay on 
I nitia l R e spons e Tim e for a ll SS a le rts, p > .05 

> Pilots on average notified ATC 5.13 seconds 
faster in the perfect sensor model condition 

■ Median Difference = 3 seconds faster in 
perfect sensor model condition 
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Stages of Pilot DAA Task 
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Configuration 

Th e r e was no significant e ff e ct of Disp l ay on 
I niti al R e spons e Tim e for a ll SS ale rts, p > .05 

Pilots took an overall average of 12.35 
seconds to initiate an edit in response to a 
Corrective SS or SS Warning alert 



> 50% of pilots initiated an edit within 12 
seconds of the onset of the alert 

> 90% of pilots initiated an edit within 36 
seconds of the onset of the alert 
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Configuration 


> Pilots on average initiated an edit 8.60 > Unable to test for significant interaction 

seconds faster following an SS Warning alert ■ Large differences between Corrective and 

than a Corrective SS alert Warning Initial RTs for all configurations 

■ Median Difference = 6 second faster in 
response to an SS Warning alert 
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> Th e r e was no significant e ff e ct of Disp l ay on 
I n i tia l R e spons e T i m e for a ll SS a le rts, p > .05 

> Pilots on average initiated an edit 1.30 seconds 
faster in response to alerts to non-cooperative 
aircraft 

■ Median Difference = 1 seconds faster in response 
to non-cooperative traffic 


> Th e r e was no significant e ff e ct of Disp l ay on 
I nitia l R e spons e Tim e for a ll SS a le rts, p > .05 

> Pilots on average initiated an edit 0.85 seconds 
faster in the noisy sensor model condition 

■ Median Difference = 1 seconds faster in the noisy 
sensor model condition 
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Th e r e was no significant e ff e ct of Disp l ay on 
I niti al R e spons e Tim e for a ll SS ale rts, p > .05 

Pilots took an overall average of 8 seconds 
to make an initial edit to a Corrective SS or 
SS Warning alert 
- Median = 6 seconds 


> 50% of pilots notified ATC within 6 seconds 
of the onset of the alert 

> 90% of pilots notified ATC within 16 seconds 
of the onset of the alert 
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02 Configuration 03 


> Th e r e was no significant e ff e ct of Disp l ay on 
I n i tia l R e spons e T i m e for a ll SS a le rts, p > .05 

> Pilots on average made initial edits 3.32 
seconds faster following an SS Warning alert 
than a Corrective SS alert 

■ Median Difference = 1 second faster in response 
to SS Warning alert 


> Unable to test for significant interaction 

■ Much l arg e r diff e r e nc e s b e tw ee n Corr e ctiv e 
Notify tim e s and Warning Notify tim e s for D1 
and DA 
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Th e r e was no significant e ff e ct of Disp l ay on 
I n i tia l R e spons e T i m e for a ll SS a le rts, p > .05 

Pilots on average made initial edits 0.43 
seconds faster in response to alerts to non - 
cooperative aircraft 

■ Median Difference = 0 seconds faster in response 
to non-cooperative traffic 


> Th e r e was no significant e ff e ct of Disp l ay on 
I n i tia l R e spons e Tim e for a ll SS a le rts, p > .05 

> Pilots on average made initial edits 1.10 
seconds faster in the noisy sensor model 
condition 

■ Median Difference = 1 seconds faster in noisy 
sensor model condition 
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Total Edit Histogram 



Th e r e was no significant e ff e ct of Disp l ay on 
Tot al R e spons e Tim e for a ll SS a le rts, p > .05 

Pilots took an overall average of 10.7 
seconds to upload their final maneuver after 
the initiation of an edit 


> 50% of pilots completed their final upload 
within 7 seconds of the initiation of an edit 

> 90% of pilots completed their final upload 
within 22 seconds of the initiation of an edit 


Median = 7 seconds 
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Basic Stratway+ OmniBands Vector Planner 

Configuration 


> Th e r e was no significant e ff e ct of Disp l ay on 
I n i tia l R e spons e T i m e for a ll SS a le rts, p > .05 

> Pilots on average made initial edits 3.32 
seconds faster following an SS Warning alert 
than a Corrective SS alert 

■ Median Difference = 1 second faster in response 
to SS Warning alert 


> Unable to test for significant interaction 

■ Much l arg e r diff e r e nc e s b e tw ee n Corr e ctiv e 
Notify tim e s and Warning Notify tim e s for D1 
and DA 
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Sensor Model 


> Th e r e was no significant e ff e ct of Disp l ay on 
I n i tia l R e spons e T i m e for a ll SS a le rts, p > .05 

> Pilots on average made initial edits 0.43 
seconds faster in response to alerts to non - 
cooperative aircraft 

■ Median Difference = 0 seconds faster in response 
to non-cooperative traffic 


> Th e r e was no significant e ff e ct of Disp l ay on 
I n i tia l R e spons e Tim e for a ll SS a le rts, p > .05 

> Pilots on average made initial edits 1.10 
seconds faster in the noisy sensor model 
condition 

■ Median Difference = 1 seconds faster in noisy 
sensor model condition 
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• Th e r e was no significant e ff e ct of Disp l ay on 
I niti al R e spons e Tim e for a ll SS ale rts, p > .05 

• Pilots took an overall average of 20 seconds 
to start a maneuver in response to a 
Corrective SS or SS Warning alert 

- Median = 18 seconds 


> 50% of pi l ots notifi e d ATC within 12 s e conds 
of th e ons e t of th e a le rt 

> 90% of pi l ots notifi e d ATC within 32 s e conds 
of th e ons e t of th e a le rt 
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Configuration 


> Th e r e was no significant e ff e ct of Disp l ay on 
I n i tia l R e spons e T i m e for a ll SS a le rts, p > .05 

> Pilots on average started a maneuver 12.05 
seconds faster following an SS Warning alert 
than a Corrective SS alert 

■ Median Difference = X second faster in starting a 
maneuver to SS Warning alert 


> Unable to test for significant interaction 

■ Much l arg e r diff e r e nc e s b e tw ee n Corr e ctiv e 
Notify tim e s and Warning Notify tim e s for D1 
and DA 
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> Th e r e was no significant e ff e ct of Disp l ay on 
I n i tia l R e spons e T i m e for a ll SS a le rts, p > .05 

> Pilots on average started a maneuver 1.66 
seconds faster in response to alerts to non- 
cooperative aircraft 

■ Median Difference = 2 seconds faster in starting 
a maneuver to non-cooperative traffic 


> Th e r e was no significant e ff e ct of Disp l ay on 
I n i tia l R e spons e Tim e for a ll SS a le rts, p > .05 

> Pilots on average completed maneuvers 0.17 
seconds faster in the noisy sensor model 
condition 

■ Median Difference = 2 seconds faster in noisy 
sensor model condition 
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Bin 


Th e r e was no significant e ff e ct of Disp l ay on 
I niti al R e spons e Tim e for a ll SS ale rts, p > .05 

Pilots took an overall average of 26.48 
seconds to eliminate a Corrective SS or SS 
Warning alert 
— Median = 22 seconds 


> 50% of pilots cleared an alert within 21 
seconds of its onset 

> 90% of pilots cleared an alert within 45 
seconds of its onset 
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D2 Configuration D3 


D4 


> Th e r e was no significant e ff e ct of Disp l ay on 
I n i tia l R e spons e T i m e for a ll SS a le rts, p > .05 

> Pilots on average cleared an alert 6.50 seconds 
faster following an SS Warning alert than a 
Corrective SS alert 

■ Median Difference = 6 second faster in response 
to an SS Warning alert 


> Unable to test for significant interaction 

■ Nothing approaching an interaction 
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Perfect Noise 


Sensor Model 


> Th e r e was no significant e ff e ct of Disp l ay on 
I n i tia l R e spons e T i m e for a ll SS a le rts, p > .05 

> Pilots on average cleared an alert 6.69 seconds 
faster in response to alerts to non-cooperative 
aircraft 

■ Median Difference = 3 seconds faster in response 
to non-cooperative traffic 


> Th e r e was no significant e ff e ct of Disp l ay on 
I nitia l R e spons e Tim e for a ll SS a le rts, p > .05 

> Pilots on average cleared an alert 1.04 seconds 
faster in the noisy sensor model condition 

■ Median Difference = 2 seconds faster in the noisy 
sensor model condition 
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D4 


Th e r e was no significant e ff e ct of Disp l ay on 
I niti al R e spons e Tim e for a ll SS ale rts, p > .05 

Pilots took an overall average of 36.77 
seconds to complete a maneuver following a 
Corrective SS or SS Warning alert 
- Median = 33 seconds 


> 50% of pi l ots notifi e d ATC within 12 s e conds 
of th e ons e t of th e a le rt 

> 90% of pi l ots notifi e d ATC within 32 s e conds 
of th e ons e t of th e a le rt 
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> Th e r e was no significant e ff e ct of Disp l ay on > Unable to test for significant interaction 
I n i tia l R e spons e T i m e for a ll SS a le rts, p > .05 

> Pilots on average completed a maneuver 13.00 
seconds faster following an SS Warning alert 
than a Corrective SS alert 

■ Median Difference = 7 seconds faster in 
completion of maneuver to SS Warning alert 
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Sensor Model 


> Th e r e was no significant e ff e ct of Disp l ay on 
I n i tia l R e spons e T i m e for a ll SS a le rts, p > .05 

> Pilots on average completed maneuvers 5.21 
seconds faster in response to alerts to non- 
cooperative aircraft 

■ Median Difference = 1 seconds faster in 

maneuver completion to non-cooperative traffic 


> Th e r e was no significant e ff e ct of Disp l ay on 
I n i tia l R e spons e Tim e for a ll SS a le rts, p > .05 

> Pilots on average completed maneuvers 0.64 
seconds faster in the noisy sensor model 
condition 

■ Median Difference = 3.5 seconds faster in noisy 
sensor model condition 


Compliance Time 



D1 

D2 

D3 

D4 

Min 

10 

12 

6 

6 

Q1 

24.75 

21 

26 

26 

Median 

34 

31 

33 

33 

Q3 

46 

38 

43 

43 

Max 

93 

85 

99 

99 


110 
100 
90 
80 
oi 70 



0 

D1 D2 D3 


Display Configuration 




Seconds 


Resume Own Navigation 


100 
90 
80 
70 
60 
50 
40 
30 
20 
10 
0 

Basic Stratway+ OmniBands Vector Planner 

Configuration 

• Th e r e was no significant e ff e ct of Disp l ay on 
I niti al R e spons e Tim e for a ll SS ale rts, p > .05 

• Pilots took an overall average of 36.77 
seconds to complete a maneuver following a 
Corrective SS or SS Warning alert 

- Median = 33 seconds 
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> 50% of pilots notifi e d ATC within 12 s e conds 
of th e ons e t of th e al e rt 

> 90% of pilots notifi e d ATC within 32 s e conds 
of th e ons e t of th e al e rt 
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Basic Stratway+ OmniBands Vector Planner 

Configuration 


> Th e r e was no significant e ff e ct of Disp l ay on > Unable to test for significant interaction 
I n i tia l R e spons e T i m e for a ll SS a le rts, p > .05 

> Pilots on average completed a maneuver 13.00 
seconds faster following an SS Warning alert 
than a Corrective SS alert 

■ Median Difference = 7 seconds faster in 
completion of maneuver to SS Warning alert 



Resume Own Navigation 


By Intruder Equipage 


By Sensor Model 
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□ Mean 
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Perfect Noise 

Sensor Model 


> Th e r e was no significant e ff e ct of Disp l ay on 
I n i tia l R e spons e T i m e for a ll SS a le rts, p > .05 

> Pilots on average completed maneuvers 5.21 
seconds faster in response to alerts to non- 
cooperative aircraft 

■ Median Difference = 1 seconds faster in 

maneuver completion to non-cooperative traffic 


> Th e r e was no significant e ff e ct of Disp l ay on 
I n i tia l R e spons e Tim e for a ll SS a le rts, p > .05 

> Pilots on average completed maneuvers 0.64 
seconds faster in the noisy sensor model 
condition 

■ Median Difference = 3.5 seconds faster in noisy 
sensor model condition 




Resume Own Navigation 



D1 

D2 

D3 

D4 

Min 

13 

17 

24 

32 

Q1 

63.5 

48 

61 

59 

Median 

78.5 

66 

75.5 

84 

Q3 

95.25 

87 

91.25 

99 

Max 

132 

158 

143 

159 


Basic Stratway+ OmniBands Vector Planner 

Display Configuration 
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NASA-TLX Summary 


Omni Bands (D3) resulted in significantly lower workload ratings than the 
Vector Planner (D4) for 5 of the 6 scales: 

- Mental, Temporal, Effort, Frustration, & Performance Degradation 

• Only Physical Demands failed to result in a significant difference 

Frustration was the only scale that resulted in more than one significant 
difference 

- Stratway Bands (D2) were also significantly less impactful than the Vector 
Planner (D4) 

- Omni Bands (D3) were also significantly less impactful than the Information 
Only condition (Dl) 


POST-TRIAL QUESTIONNAIRE 

BY DISPLAY CONFIGURATION 


Post Trial 


Question 1 

— Rate your ability to maintain separation from other aircraft 

(p < 0.001) 

• Difference between D1 and D3 (p < 0.001) 

• Difference between D3 and D4 (p = 0.006) 


Excellent 5 

4.5 

4 

3.5 
3 

2.5 
2 

1.5 
1 

0.5 

Unacceptable 0 



Post Trial 


• Question 2 

- Rate your ability to minimize deviations from the planned 
path (p < 0.001) 

• Difference between D1 and D2 (p = 0.003) 

• Difference between D1 and D3 (p = 0.003) 



D1 D2 D3 D4 


Post Trial 


• Question 3 

- Rate your ability to handle all pilot tasks ( p = 0.001) 

• Difference between D1 and D2 (p = 0.039) 

• Difference between D1 and D3 (p = 0.039) 

• Difference between D2 and D4 (p = 0.034) 

• Difference between D3 and D4 (p = 0.054) 


Excellent 5 



Post Trial 


• Question 4 

- Rate the complexity of the encounters in this trial (p 
< 0 . 001 ) 

• Difference between D1 and D3 (p = 0.011) 

• Difference between D3 and D4 (p < 0.001) 


Very Difficult 5 


4.5 
4 

3.5 



Post Trial 


• Question 6 

- This display was easy to use (p = 0.001) 

• Difference between D1 and D3 (p = 0.035) 

• Difference between D1 and D4 (p = 0.050) 

• Difference between D2 and D3 (p = 0.020) 

• Difference between D3 and D4 (p = 0.001) 


Strongly Agree 5 

4.5 
4 

3.5 
3 

2.5 
2 

1.5 
1 

0.5 

Strongly Disagree o 



Post Trial 


Question 7 

— This display was easy to understand (Marginal, p = 0.069) 

• Difference between D3 and D4 



Post Trial 


• Question 8 

- This display provided the necessary information to predict 
a potential loss of separation (p < 0.001) 

• Difference between D1 and D2 (p = 0.009) 

• Difference between D1 and D3 (p < 0.001) 

• Difference between D3 and D4 (p = 0.002) 



D1 


D2 


D3 


D4 


Post Trial 


• Question 9 

- This display provided the necessary information to 
perform a maneuver for separation (p < 0.001) 

• Difference between D1 and D2 (p = 0.005) 

• Difference between D1 and D3 (p < 0.001) 

• Difference between D2 and D4 (p = 0.037) 

• Difference between D3 and D4 (p = 0.003) 



D1 D2 D3 D4 


Post Trial 


• Question 10 

- I trusted the accuracy of the information provided by the 
display (p = NS) 



Post Trial 


• Question 11 

— I felt I had enough training with this display to be able to 
operate it safely during this trial ( p = NS) 



D1 D2 D3 D4 


POST-SIM QUESTIONNAIRE 


Visual Alert Levels 




Preventive Self 
Separation Alert 

Corrective Self 
Separation Alert 

Self Separation 
Warning Alert 

1 . The visual display of this alert 
(i.e., icon color, shape, etc.) 
was easy to understand 

M = 4.44 
SE = 0.16 

M = 4.50 
SE = 0.13 

M = 5.0 
SE = 0.0 

2. Based on this alert, I would 
likely contact ATC and then 
maneuver 

M = 2.50 
SE = 0.38 

M = 3.81 
SE = 0.31 

M = 1.38 
SE = 0.26 

3. Based on this alert, I would 
likely maneuver prior to 
contacting ATC 

M = 1.56 
SE = 0.20 

M = 2.63 
SE = 0.27 

M = 5.00 
SE = 0.0 


Scale 

1 

Strongly Disagree 

2 

Somewhat Disagree 

3 

Neither Agree nor 
Disagree 

4 

Somewhat Agree 

5 

Strongly Agree 






Question 4 


Asked pilots if they understood which of the 3 alerts would 
lead to a LoWC if ownship and intruder trajectories remained 
unchanged 

- 14/16 pilots correctly reported that Corrective SS and SS Warning 
alerts were predicted to lead to LoWC & Preventive SS were not 

- 1 pilot incorrectly reported that a Preventive SS was predicted to lead 
to LoWC 

- 1 pilot incorrectly reported that Corrective SS would not lead to LoWC 

In debrief, all 16 pilots demonstrated that they correctly 
understood alerting structure 


Auditory Alert Levels 




“Traffic, 
Monitor ” 

“Traffic, 
Separate ” 

“Traffic, 

Maneuver 

Now” 

5. This auditory alert was 
clearly distinguishable from 
the other auditory alerts 

M = 4.13 
SE = 0.24 

M =4.25 
SE = 0.21 

M = 4.68 
SE = 0.18 

6. This auditory alert was useful 
for maintaining separation 

M = 3.75 
SE = 0.23 

M =4.31 
SE = 0.20 

M = 4.68 
SE = 0.15 


Scale 

1 

Strongly Disagree 

2 

Somewhat Disagree 

3 

Neither Agree nor 
Disagree 

4 

Somewhat Agree 

5 

Strongly Agree 






Alert Timing 


■ 

ItlolBl 

Preventive Self 
Separation Alert 

Corrective Self 
Separation Alert 

Self Separation 
Warning Alert 

7. The onset of the alert was: 

M = 3.00 
SE = 0.0 

M = 3.06 
SE = 0.06 

M = 3.13 
SE = 0.13 


Scale 

1 

Too Early 

2 

Somewhat Early 

3 

Appropriate 

4 

Somewhat Late 

5 

Too Late 






Questions 8-10 


• Compared to the information only display, how did the 
other three displays affect your ability to maintain 
separations? 


Significantly Better 5 
Than D1 4.5 



Question 11 


• Rank the following display configurations in order of the 
effects on your ability to maintain separation 
- 1 st = Best Supported; 4 th = Least Supported 


Rank 

D1 

D2 

D3 

D4 

1st 

0/16 

2/16 

14/16 

0/16 

2nd 

1/16 

10/16 

1/16 

4/16 

3rd 

8/16 

3/16 

1/16 

4/16 

4th 

7/16 

1/16 

0/16 

8/16 

Avg. 

3.38 

2.19 

1.19 

3.25 








Additional Questions 


Question 12: 

- Did you notice any noise in the position data for aircraft within 
range? 

• 4/8 pilots reported ‘Yes’ (i.e., Vi of the pilots that received the 
noisy condition) 

Question 13: 

- Did you want to know which sensor was tracking the target 

• 7/16 of the pilots reported ‘Yes’ 

Question 14: 

- Do you think non-cooperative traffic need different visual or 
auditory alerting than cooperative traffic? 

• 5/16 of the pilots reported 'Yes' 


Additional Questions 


Question 15: 

- Was the surveillance volume/range sufficient to maintain 
separation from: 

• Cooperative traffic: All 16 pilots reported ‘Yes’ 

• Non-Cooperative traffic: All 16 pilots reported ‘Yes’ 

Question 16: 

- The flight ID was useful with non-cooperative traffic? 

• m = 4.375 (Slightly/Strongly Agree) 

Question 17: 

- Did you want to see the flight ID with non-cooperative 
traffic 

• 11/16 pilots reported 'Yes' 


Question 18 


Did any of the following factors influence whether you 
decided to make a horizontal maneuver? Please circle all 
relevant factors. 

— Mission Profile 11/16 

— Whether or not ownship was in level flight 8/16 

— Encounter Geometry 12/16 

— Uncertainty 7/16 

— Other* 7/16 

— None *"Other" answers^nojigpilots: 

- Visual cue of non-overlapping AC, feel safer 

- Knowing where the other AC was leveling off 

- Speed of AC/Closure Rate 

- ATC preference for maintaining altitude 

- Amount of anticipated deviation 


Question 19 


Did any of the following factors influence whether you 
decided to make a vertical maneuver? Please circle all 


relevant factors. 

— Mission Profile 8/16 

— Whether or not ownship was in level flight 7/16 

— Encounter Geometry 10/16 

— Uncertainty 4/16 

— Other 7/17 

— None 2/16 


*"Other" answers from pilots: 

- Whether the intruder was climbing or descending 

- Knowing where the other AC was leveling off 

- Speed of AC/Closure Rate 

- Follow on manipulation of lost/link emergency 

- Time to return from deviation 

- Alt changes are different in RPAs too much to 
change 


END-OF-SIM DEBRIEF 


Debrief 


• Question 1 

- Do you view proximal alerts as a "pre alert" or as an 
indication of proximity? 

• Proximity 5/16 

• Pre-alert 7/16 

• Both 2/16 

• General awareness 1/16 

• Didn't attend to or notice them 1/16 


Debrief 


Question 2 

- Did you clearly understand the meaning of the 4 
different alerts? 

• 16 clearly understood 

• 1 pilot stated corrective should be red since it expects you to take 
action 


Debrief 


Question 3 

- Did you find the meaning/intent of the 4 different alerts 
distinct from one another 

• 16 pilots said yes 

- All pilots liked the alerting system and thought it was clear and logically 
progressive 

• However, 5/16 pilots reported that the least meaningful/useful difference was 
between the corrective and warning alerts - e.g., "if I have to maneuver, it 
should be red/a warning" 


Debrief 


Question 4 

- Did you find the auditory alerts easy to 
understand? If no, what could be improved upon 

• 15 pilots said yes 

- 4 pilots said they treated audio more as a cue to check out visual 

• 1 pilot said the difference between 'separate' and 'maneuver now' 
did not seem clear 


Debrief 


Question 5 

- Did you feel that the auditory alerts complimented 
the associated meaning of the alerts 

• 14/16 pilots thought they complimented the associated meaning 

• 1 pilot found 'separate' and 'monitor' ambiguous 

• 1 pilot said he liked "maneuver now" the most 


Debrief 


Question 6 

- Did you use any information not provided in the 
TSD to inform your eventual maneuver? 

• 6/16 pilots looked at Intruder (encounter) geometry 

• 6/16 pilots looked at whether the intruders were climbing/descending 
around their climbs and descents 

• 2/16 pilots said previous experience 

• 2/16 pilots said minimizing course deviation 


Debrief 


Question 7 

- Did the separation maneuver recommendations 
usually correspond to what you preferred to do? 

• 5/16 said they were similar maneuvers - i.e., the guidance was 
recommending maneuvers that made sense to them 

• 6/16 pilots didn't like the trial planner (having to activate, etc.) 

• 3/16 didn't like the Stratway bands because didn't tell which alert 
would occur 

• 1 pilot didn't like how the algorithms were only trying to minimize 
deviation size - would prefer to maneuver larger to eliminate the 
conflict more quickly 

• 1 pilot didn't like the Omni bands going all red 


Debrief 


Question 8 

- "Did you find any of the alerts unnecessary?” 

• 13/15 no, all necessary or all served a purpose 

• 1/15 said does not absolutely need the corrective - don't always need 
to accommodate ATC, can survive just on the warning/maneuver now 

• 1/15 said proximal and preventive were very similar - pay attention 
but no maneuver needed 


Debrief 


• Question 9 

- “Was there any additional information you would 
want to see?” 

• 6/15 said nothing else needed (all there) 

• 4/15 said vertical trend arrow in data tag 

• 3/15 said dead reckoning line 

• 1 said steering window dockable at all times 

• 1 said what to do like a TA/RA 


Debrief 


Question 10 

- "What are the minimum horizontal and vertical 
distances you would comfortably pass by an 
intruder"? 

• 10/16 - 500ft .75nm 

• 1/16- 1000ft lnm 

• 2/16 - 1000ft ,75nm 

• 3/16 - Depends on the Aircraft 


BACK-UP SLIDES 


NASA-TLX 




• Mental (p = .001) 

• Difference between D3 and D4 (p < .001) 


NASA-TLX Mental Scale 



D1 D2 D3 D4 


NASA-TLX 


Physical (A/S) 


NASA-TLX Physical Scale 



NASA-TLX 


• Temporal (marginal, p = 0.065) 

• Difference between D3 and D4 (p = 0.023) 



0 


D1 


D2 


D3 


D4 


NASA-TLX 


• Effort (p = 0.004) 

• Difference between D3 and D4 (p = 0.012) 


NASA-TLX Effort Scale 



D1 D2 D3 D4 


NASA-TLX 


• Frustration (p < 0.001) 

• Difference between D1 and D3 (p = 0.021) 

• Difference between D2 and D4 (p = 0.017) 

• Difference between D3 and D4 (p = 0.010) 


NASA-TLX Frustration Scale 



D1 D2 D3 D4 


NASA-TLX 


• Degradation (p = 0.001) 

• Difference between D3 and D4 (p = 0.034) 


NASA-TLX Degradation Scale 



